Establishing a reference focal plane using convolutional neural networks and beads for brightfield imaging.

Repeatability of measurements from image analytics is difficult, due to the heterogeneity and complexity of cell samples, exact microscope stage positioning, and slide thickness. We present a method to define and use a reference focal plane that provides repeatable measurements with very high accuracy, by relying on control beads as reference material and a convolutional neural network focused on the control bead images. Previously we defined a reference effective focal plane (REFP) based on the image gradient of bead edges and three specific bead image features. This paper both generalizes and improves on this previous work. First, we refine the definition of the REFP by fitting a cubic spline to describe the relationship between the distance from a bead's center and pixel intensity and by sharing information across experiments, exposures, and fields of view. Second, we remove our reliance on image features that behave differently from one instrument to another. Instead, we apply a convolutional regression neural network (ResNet 18) trained on cropped bead images that is generalizable to multiple microscopes. Our ResNet 18 network predicts the location of the REFP with only a single inferenced image acquisition that can be taken across a wide range of focal planes and exposure times. We illustrate the different strategies and hyperparameter optimization of the ResNet 18 to achieve a high prediction accuracy with an uncertainty for every image tested coming within the microscope repeatability measure of 7.5 µm from the desired focal plane. We demonstrate the generalizability of this methodology by applying it to two different optical systems and show that this level of accuracy can be achieved using only 6 beads per image.

High-volume, label-free imaging for quantifying single-cell dynamics in induced pluripotent stem cell colonies.

To facilitate the characterization of unlabeled induced pluripotent stem cells (iPSCs) during culture and expansion, we developed an AI pipeline for nuclear segmentation and mitosis detection from phase contrast images of individual cells within iPSC colonies. The analysis uses a 2D convolutional neural network (U-Net) plus a 3D U-Net applied on time lapse images to detect and segment nuclei, mitotic events, and daughter nuclei to enable tracking of large numbers of individual cells over long times in culture. The analysis uses fluorescence data to train models for segmenting nuclei in phase contrast images. The use of classical image processing routines to segment fluorescent nuclei precludes the need for manual annotation. We optimize and evaluate the accuracy of automated annotation to assure the reliability of the training. The model is generalizable in that it performs well on different datasets with an average F1 score of 0.94, on cells at different densities, and on cells from different pluripotent cell lines. The method allows us to assess, in a non-invasive manner, rates of mitosis and cell division which serve as indicators of cell state and cell health. We assess these parameters in up to hundreds of thousands of cells in culture for more than 36 hours, at different locations in the colonies, and as a function of excitation light exposure.

Three-dimensional, label-free cell viability measurements in tissue engineering scaffolds using optical coherence tomography.

In the field of tissue engineering, 3D scaffolds and cells are often combined to yield constructs that are used as therapeutics to repair or restore tissue function in patients. Viable cells are often required to achieve the intended mechanism of action for the therapy, where the live cells may build new tissue or may release factors that induce tissue regeneration. Thus, there is a need to reliably measure cell viability in 3D scaffolds as a quality attribute of a tissue-engineered medical product. Here, we developed a noninvasive, label-free, 3D optical coherence tomography (OCT) method to rapidly (2.5 min) image large sample volumes (1 mm3 ) to assess cell viability and distribution within scaffolds. OCT imaging was assessed using a model scaffold-cell system consisting of a polysaccharide-based hydrogel seeded with human Jurkat cells. Four test systems were used: hydrogel seeded with live cells, hydrogel seeded with heat-shocked or fixed dead cells and hydrogel without any cells. Time series OCT images demonstrated changes in the time-dependent speckle patterns due to refractive index (RI) variations within live cells that were not observed for pure hydrogel samples or hydrogels with dead cells. The changes in speckle patterns were used to generate live-cell contrast by image subtraction. In this way, objects with large changes in RI were binned as live cells. Using this approach, on average, OCT imaging measurements counted 326 ± 52 live cells per 0.288 mm3 for hydrogels that were seeded with 288 live cells (as determined by the acridine orange-propidium iodide cell counting method prior to seeding cells in gels). Considering the substantial uncertainties in fabricating the scaffold-cell constructs, such as the error from pipetting and counting cells, a 13% difference in the live-cell count is reasonable. Additionally, the 3D distribution of live cells was mapped within a hydrogel scaffold to assess the uniformity of their distribution across the volume. Our results demonstrate a real-time, noninvasive method to rapidly assess the spatial distribution of live cells within a 3D scaffold that could be useful for assessing tissue-engineered medical products.

Measuring dimensionality of cell-scaffold contacts of primary human bone marrow stromal cells cultured on electrospun fiber scaffolds.

The properties and structure of the cellular microenvironment can influence cell behavior. Sites of cell adhesion to the extracellular matrix (ECM) initiate intracellular signaling that directs cell functions such as proliferation, differentiation, and apoptosis. Electrospun fibers mimic the fibrous nature of native ECM proteins and cell culture in fibers affects cell shape and dimensionality, which can drive specific functions, such as the osteogenic differentiation of primary human bone marrow stromal cells (hBMSCs), by. In order to probe how scaffolds affect cell shape and behavior, cell-fiber contacts were imaged to assess their shape and dimensionality through a novel approach. Fluorescent polymeric fiber scaffolds were made so that they could be imaged by confocal fluorescence microscopy. Fluorescent polymer films were made as a planar control. hBSMCs were cultured on the fluorescent substrates and the cells and substrates were imaged. Two different image analysis approaches, one having geometrical assumptions and the other having statistical assumptions, were used to analyze the 3D structure of cell-scaffold contacts. The cells cultured in scaffolds contacted the fibers in multiple planes over the surface of the cell, while the cells cultured on films had contacts confined to the bottom surface of the cell. Shape metric analysis indicated that cell-fiber contacts had greater dimensionality and greater 3D character than the cell-film contacts. These results suggest that cell adhesion site-initiated signaling could emanate from multiple planes over the cell surface during culture in fibers, as opposed to emanating only from the cell's basal surface during culture on planar surfaces.

Label-free cell tracking enables collective motion phenotyping in epithelial monolayers.

Collective cell migration is an umbrella term for a rich variety of cell behaviors, whose distinct character is important for biological function, notably for cancer metastasis. One essential feature of collective behavior is the motion of cells relative to their immediate neighbors. We introduce an AI-based pipeline to segment and track cell nuclei from phase-contrast images. Nuclei segmentation is based on a U-Net convolutional neural network trained on images with nucleus staining. Tracking, based on the Crocker-Grier algorithm, quantifies nuclei movement and allows for robust downstream analysis of collective motion. Because the AI algorithm required no new training data, our approach promises to be applicable to and yield new insights for vast libraries of existing collective motion images. In a systematic analysis of a cell line panel with oncogenic mutations, we find that the collective rearrangement metric, D2 min, which reflects non-affine motion, shows promise as an indicator of metastatic potential.

QUAREP-LiMi: A community-driven initiative to establish guidelines for quality assessment and reproducibility for instruments and images in light microscopy.

A modern day light microscope has evolved from a tool devoted to making primarily empirical observations to what is now a sophisticated , quantitative device that is an integral part of both physical and life science research. Nowadays, microscopes are found in nearly every experimental laboratory. However, despite their prevalent use in capturing and quantifying scientific phenomena, neither a thorough understanding of the principles underlying quantitative imaging techniques nor appropriate knowledge of how to calibrate, operate and maintain microscopes can be taken for granted. This is clearly demonstrated by the well-documented and widespread difficulties that are routinely encountered in evaluating acquired data and reproducing scientific experiments. Indeed, studies have shown that more than 70% of researchers have tried and failed to repeat another scientist's experiments, while more than half have even failed to reproduce their own experiments. One factor behind the reproducibility crisis of experiments published in scientific journals is the frequent underreporting of imaging methods caused by a lack of awareness and/or a lack of knowledge of the applied technique. Whereas quality control procedures for some methods used in biomedical research, such as genomics (e.g. DNA sequencing, RNA-seq) or cytometry, have been introduced (e.g. ENCODE), this issue has not been tackled for optical microscopy instrumentation and images. Although many calibration standards and protocols have been published, there is a lack of awareness and agreement on common standards and guidelines for quality assessment and reproducibility. In April 2020, the QUality Assessment and REProducibility for instruments and images in Light Microscopy (QUAREP-LiMi) initiative was formed. This initiative comprises imaging scientists from academia and industry who share a common interest in achieving a better understanding of the performance and limitations of microscopes and improved quality control (QC) in light microscopy. The ultimate goal of the QUAREP-LiMi initiative is to establish a set of common QC standards, guidelines, metadata models and tools, including detailed protocols, with the ultimate aim of improving reproducible advances in scientific research. This White Paper (1) summarizes the major obstacles identified in the field that motivated the launch of the QUAREP-LiMi initiative; (2) identifies the urgent need to address these obstacles in a grassroots manner, through a community of stakeholders including, researchers, imaging scientists, bioimage analysts, bioimage informatics developers, corporate partners, funding agencies, standards organizations, scientific publishers and observers of such; (3) outlines the current actions of the QUAREP-LiMi initiative and (4) proposes future steps that can be taken to improve the dissemination and acceptance of the proposed guidelines to manage QC. To summarize, the principal goal of the QUAREP-LiMi initiative is to improve the overall quality and reproducibility of light microscope image data by introducing broadly accepted standard practices and accurately captured image data metrics.

Establishing a reference focal plane using beads for trypan-blue-based viability measurements.

Trypan blue dye exclusion-based cell viability measurements are highly dependent upon image quality and consistency. In order to make measurements repeatable, one must be able to reliably capture images at a consistent focal plane, and with signal-to-noise ratio within appropriate limits to support proper execution of image analysis routines. Imaging chambers and imaging systems used for trypan blue analysis can be inconsistent or can drift over time, leading to a need to assure the acquisition of images prior to automated image analysis. Although cell-based autofocus techniques can be applied, the heterogeneity and complexity of the cell samples can make it difficult to assure the effectiveness, repeatability and accuracy of the routine for each measurement. Instead of auto-focusing on cells in our images, we add control beads to the images, and use them to repeatedly return to a reference focal plane. We use bead image features that have stable profiles across a wide range of focal values and exposure levels. We created a predictive model based on image quality features computed over reference datasets. Because the beads have little variation, we can determine the reference plane from bead image features computed over a single-shot image and can reproducibly return to that reference plane with each sample. The achieved accuracy (over 95%) is within the limits of the actuator repeatability. We demonstrate that a small number of beads (less than 3 beads per image) is needed to achieve this accuracy. We have also developed an open-source Graphical User Interface called Bead Benchmarking-Focus And Intensity Tool (BB-FAIT) to implement these methods for a semi-automated cell viability analyser.

It is critical for the manufacturing and release of living cell-based therapies to determine the viability, the ratio of living cells to the total number of cells (live and dead), in the therapy. Dead cells can be a safety concern for the patient, and dosing is often based on the number of living cells which are the active ingredient of the drug product. Currently, the most common approach to evaluating cell viability is based on the staining of cell samples with the trypan blue marker of cell membrane integrity: a loss in cell membrane integrity with cell death allows the dye into the cell, which can be seen using brightfield microscopy. To classify cells as live/dead, the brightness of the cells is evaluated and cells with bright centres are considered live, while those with dark centres are considered dead. Unfortunately, this approach of staining, imaging and classification is very sensitive to image acquisition settings, including image focus and brightness. This paper introduces a method to establish the required image quality for image viability analysis, providing a tool to return to image acquisition settings that will ensure image quality even when there is variability from sample to sample. In this method, polymeric beads are added to each cell sample prior to cell viability analysis. Using image processing, we extract key features from the beads in the image such as sharpness of the edges of the beads. The image features of the cells can vary significantly from sample to sample and under different cell conditions, but image features of beads have proved to be consistent across samples. We are thus able to collect reference datasets quantifying bead features over a wide range of image acquisition settings (brightness and focus), allowing us to establish a reference focal plan for image acquisition for any cell sample based on bead features. We show that with as few as three beads per image, the reference focal plane can be found from a single acquisition of beads image data over a wide range of image focuses and brightness, allowing users to consistently acquire images for cell viability that meet pre-defined quality requirements.

Deep learning predicts function of live retinal pigment epithelium from quantitative microscopy.

Increases in the number of cell therapies in the preclinical and clinical phases have prompted the need for reliable and noninvasive assays to validate transplant function in clinical biomanufacturing. We developed a robust characterization methodology composed of quantitative bright-field absorbance microscopy (QBAM) and deep neural networks (DNNs) to noninvasively predict tissue function and cellular donor identity. The methodology was validated using clinical-grade induced pluripotent stem cell-derived retinal pigment epithelial cells (iPSC-RPE). QBAM images of iPSC-RPE were used to train DNNs that predicted iPSC-RPE monolayer transepithelial resistance, predicted polarized vascular endothelial growth factor (VEGF) secretion, and matched iPSC-RPE monolayers to the stem cell donors. DNN predictions were supplemented with traditional machine-learning algorithms that identified shape and texture features of single cells that were used to predict tissue function and iPSC donor identity. These results demonstrate noninvasive cell therapy characterization can be achieved with QBAM and machine learning.

MIST: Accurate and Scalable Microscopy Image Stitching Tool with Stage Modeling and Error Minimization.

Automated microscopy can image specimens larger than the microscope's field of view (FOV) by stitching overlapping image tiles. It also enables time-lapse studies of entire cell cultures in multiple imaging modalities. We created MIST (Microscopy Image Stitching Tool) for rapid and accurate stitching of large 2D time-lapse mosaics. MIST estimates the mechanical stage model parameters (actuator backlash, and stage repeatability 'r') from computed pairwise translations and then minimizes stitching errors by optimizing the translations within a (4r)2 square area. MIST has a performance-oriented implementation utilizing multicore hybrid CPU/GPU computing resources, which can process terabytes of time-lapse multi-channel mosaics 15 to 100 times faster than existing tools. We created 15 reference datasets to quantify MIST's stitching accuracy. The datasets consist of three preparations of stem cell colonies seeded at low density and imaged with varying overlap (10 to 50%). The location and size of 1150 colonies are measured to quantify stitching accuracy. MIST generated stitched images with an average centroid distance error that is less than 2% of a FOV. The sources of these errors include mechanical uncertainties, specimen photobleaching, segmentation, and stitching inaccuracies. MIST produced higher stitching accuracy than three open-source tools. MIST is available in ImageJ at isg.nist.gov.

Lineage mapper: A versatile cell and particle tracker.

The ability to accurately track cells and particles from images is critical to many biomedical problems. To address this, we developed Lineage Mapper, an open-source tracker for time-lapse images of biological cells, colonies, and particles. Lineage Mapper tracks objects independently of the segmentation method, detects mitosis in confluence, separates cell clumps mistakenly segmented as a single cell, provides accuracy and scalability even on terabyte-sized datasets, and creates division and/or fusion lineages. Lineage Mapper has been tested and validated on multiple biological and simulated problems. The software is available in ImageJ and Matlab at isg.nist.gov.

Large-scale time-lapse microscopy of Oct4 expression in human embryonic stem cell colonies.

Identification and quantification of the characteristics of stem cell preparations is critical for understanding stem cell biology and for the development and manufacturing of stem cell based therapies. We have developed image analysis and visualization software that allows effective use of time-lapse microscopy to provide spatial and dynamic information from large numbers of human embryonic stem cell colonies. To achieve statistically relevant sampling, we examined >680 colonies from 3 different preparations of cells over 5days each, generating a total experimental dataset of 0.9 terabyte (TB). The 0.5 Giga-pixel images at each time point were represented by multi-resolution pyramids and visualized using the Deep Zoom Javascript library extended to support viewing Giga-pixel images over time and extracting data on individual colonies. We present a methodology that enables quantification of variations in nominally-identical preparations and between colonies, correlation of colony characteristics with Oct4 expression, and identification of rare events.

Enabling Interactive Measurements from Large Coverage Microscopy.

Microscopy could be an important tool for characterizing stem cell products if quantitative measurements could be collected over multiple spatial and temporal scales. With the cells changing states over time and being several orders of magnitude smaller than cell products, modern microscopes are already capable of imaging large spatial areas, repeat imaging over time, and acquiring images over several spectra. However, characterizing stem cell products from such large image collections is challenging because of data size, required computations, and lack of interactive quantitative measurements needed to determine release criteria. We present a measurement web system consisting of available algorithms, extensions to a client-server framework using Deep Zoom, and the configuration know-how to provide the information needed for inspecting the quality of a cell product. The cell and other data sets are accessible via the prototype web-based system at http://isg.nist.gov/deepzoomweb.

Survey statistics of automated segmentations applied to optical imaging of mammalian cells.

BACKGROUND: The goal of this survey paper is to overview cellular measurements using optical microscopy imaging followed by automated image segmentation. The cellular measurements of primary interest are taken from mammalian cells and their components. They are denoted as two- or three-dimensional (2D or 3D) image objects of biological interest. In our applications, such cellular measurements are important for understanding cell phenomena, such as cell counts, cell-scaffold interactions, cell colony growth rates, or cell pluripotency stability, as well as for establishing quality metrics for stem cell therapies. In this context, this survey paper is focused on automated segmentation as a software-based measurement leading to quantitative cellular measurements. METHODS: We define the scope of this survey and a classification schema first. Next, all found and manually filteredpublications are classified according to the main categories: (1) objects of interests (or objects to be segmented), (2) imaging modalities, (3) digital data axes, (4) segmentation algorithms, (5) segmentation evaluations, (6) computational hardware platforms used for segmentation acceleration, and (7) object (cellular) measurements. Finally, all classified papers are converted programmatically into a set of hyperlinked web pages with occurrence and co-occurrence statistics of assigned categories. RESULTS: The survey paper presents to a reader: (a) the state-of-the-art overview of published papers about automated segmentation applied to optical microscopy imaging of mammalian cells, (b) a classification of segmentation aspects in the context of cell optical imaging, (c) histogram and co-occurrence summary statistics about cellular measurements, segmentations, segmented objects, segmentation evaluations, and the use of computational platforms for accelerating segmentation execution, and (d) open research problems to pursue. CONCLUSIONS: The novel contributions of this survey paper are: (1) a new type of classification of cellular measurements and automated segmentation, (2) statistics about the published literature, and (3) a web hyperlinked interface to classification statistics of the surveyed papers at https://isg.nist.gov/deepzoomweb/resources/survey/index.html.

From Image Tiles to Web-Based Interactive Measurements in One Stop.

This article introduces readers to a web-based solution useful for interactive nanoscale measurements of centimeter-sized specimens. This solution is a client-server system that promotes collaborative measurements and discovery. The system consists of multiple computational modules that enable uploading microscopy images, extracting metadata, assembling many nanometer-resolution images into an image covering a centimeter-sized area, and interactive viewing and measuring of objects of interest at multiple length scales over terabyte-sized images. We illustrate the use of the system on images of aerosolized nanoparticles and dye particles on printing paper.

FogBank: a single cell segmentation across multiple cell lines and image modalities.

BACKGROUND: Many cell lines currently used in medical research, such as cancer cells or stem cells, grow in confluent sheets or colonies. The biology of individual cells provide valuable information, thus the separation of touching cells in these microscopy images is critical for counting, identification and measurement of individual cells. Over-segmentation of single cells continues to be a major problem for methods based on morphological watershed due to the high level of noise in microscopy cell images. There is a need for a new segmentation method that is robust over a wide variety of biological images and can accurately separate individual cells even in challenging datasets such as confluent sheets or colonies. RESULTS: We present a new automated segmentation method called FogBank that accurately separates cells when confluent and touching each other. This technique is successfully applied to phase contrast, bright field, fluorescence microscopy and binary images. The method is based on morphological watershed principles with two new features to improve accuracy and minimize over-segmentation. First, FogBank uses histogram binning to quantize pixel intensities which minimizes the image noise that causes over-segmentation. Second, FogBank uses a geodesic distance mask derived from raw images to detect the shapes of individual cells, in contrast to the more linear cell edges that other watershed-like algorithms produce. We evaluated the segmentation accuracy against manually segmented datasets using two metrics. FogBank achieved segmentation accuracy on the order of 0.75 (1 being a perfect match). We compared our method with other available segmentation techniques in term of achieved performance over the reference data sets. FogBank outperformed all related algorithms. The accuracy has also been visually verified on data sets with 14 cell lines across 3 imaging modalities leading to 876 segmentation evaluation images. CONCLUSIONS: FogBank produces single cell segmentation from confluent cell sheets with high accuracy. It can be applied to microscopy images of multiple cell lines and a variety of imaging modalities. The code for the segmentation method is available as open-source and includes a Graphical User Interface for user friendly execution.

Cell cycle dependent TN-C promoter activity determined by live cell imaging.

The extracellular matrix protein tenascin-C plays a critical role in development, wound healing, and cancer progression, but how it is controlled and how it exerts its physiological responses remain unclear. By quantifying the behavior of live cells with phase contrast and fluorescence microscopy, the dynamic regulation of TN-C promoter activity is examined. We employ an NIH 3T3 cell line stably transfected with the TN-C promoter ligated to the gene sequence for destabilized green fluorescent protein (GFP). Fully automated image analysis routines, validated by comparison with data derived from manual segmentation and tracking of single cells, are used to quantify changes in the cellular GFP in hundreds of individual cells throughout their cell cycle during live cell imaging experiments lasting 62 h. We find that individual cells vary substantially in their expression patterns over the cell cycle, but that on average TN-C promoter activity increases during the last 40% of the cell cycle. We also find that the increase in promoter activity is proportional to the activity earlier in the cell cycle. This work illustrates the application of live cell microscopy and automated image analysis of a promoter-driven GFP reporter cell line to identify subtle gene regulatory mechanisms that are difficult to uncover using population averaged measurements.

Overlap-Based Cell Tracker.

In order to facilitate the extraction of quantitative data from live cell image sets, automated image analysis methods are needed. This paper presents an introduction to the general principle of an overlap cell tracking software developed by the National Institute of Standards and Technology (NIST). This cell tracker has the ability to track cells across a set of time lapse images acquired at high rates based on the amount of overlap between cellular regions in consecutive frames. It is designed to be highly flexible, requires little user parameterization, and has a fast execution time.